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Abstract 

MOOCs are a recent phenomenon, although given their impact, they have been subject to a large 
debate. Several questions have been raised by researchers and educators alike regarding their 
sustainability, both economically and as an efficient mode of education provision. In this paper 
we contribute to this discussion by presenting a case study of the MOOC on Lived Experiences of 
Climate Change, which piloted the iMOOC pedagogical model developed at Universidade Aberta 
(UAb), the Portugese Distance Learning University. The iMOOC is a hybrid model which 
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incorporates elements from existing MOOCs but adds other features drawn from UAb's 
experience with online learning and aims at better integrating in the larger context of the 
institutional pedagogical culture. The iMOOC implied also an integration of platforms - Moodle 
and Elgg. The pilot course had more than one thousand registrations, and it was the largest 
MOOC course on Portuguese language delivered so far. We discuss the effort required to design 
and deliver the course, the technological solution developed, and the results obtained. We 
registered a moderate effort to create and run the course, ensured by internal staff from the 
University. The technological solution was a success: an integrated architecture combining well- 
established, well-tested open software. The completion rate was 3.3%, but the high success of this 
innovative learning experience was demonstrated by the active involvement of about 50% of the 
registered participants, that followed the course until the end. Lessons learned from this 
experience and future research on the field are also discussed. 

Keywords: Massive Open Online Course, iMOOC, Pedagogical Model, Learning Effectiveness, 
Completion Rate, Cost Analysis. 


Introduction 

Massive Open Online Courses (MOOCs) were originated from the success of the now historical 
“An Introduction to AI” course designed by Sebastian Thrun and Peter Norvig at Stanford 
University in 2011 (Rodriguez, 2012). This event set in motion what would become the 
educational phenomenon of 2012 (Daniel, 2012). MOOCs can be best defined as courses designed 
for large numbers of participants that can be accessed by anyone anywhere as long as they have 
an internet connection, are open to everyone without entry qualifications, and offer a 
full/complete course experience online for free (Brouns et al., 2014). Although most MOOCs 
consist of relatively short video lectures and related content with feedback being managed either 
with peer-review and group collaboration or by automation (Alraimi, Zo & Ciganek, 2015), others 
provide a study guide or syllabus, include limited interaction with academic staff, have authentic 
activities and tests, and offer recognition options [formal and non-formal] (Brouns et al., 2014). 

Literature recognizes two main types of MOOCs: Connectivist Massive Open Online Courses 
(cMOOCs) and extended Massive Open Online Courses (xMOOCs). cMOOCs are focused on the 
learning community and connections between members of the community across the web, rather 
than on course content or the instructor, and have been carried out with great success (Rodriguez, 
2012). On the other hand, xMOOCs are based on scalability of provision, i.e., they are focused on 
giving many students access to an online course within the same platform. The style of learning is 
also different; xMOOCs tend to privilege individual studying, while cMOOCs focus on networked 
learning across several web tools and services (Conole, 2013, Rodriguez, 2012, Hew & Cheung, 
2014). As George Siemens summarizes: cMOOCs focus on knowledge creation and generation 
whereas xMOOCs focus on knowledge duplication (Siemens, 2012). However, recently there's a 
trend, especially in Europe, towards the emergence of hybrid approaches which tiy to mix the 
potential of open socially-driven learning with structured learning paths. In fact, the disruptive 

This work is licensed under a Creative Commons Attribution 4.0 International License. 


153 



iMOOC on Climate Change: Evaluation of a Massive Open Online Learning Pilot Experience 
Rocio, Coelho, Caeiro, Nicolau and Teixeira 

nature of the connectivist approach to learning is quite difficult to articulate with typical 
institutional environments (Hernandez, Morales, Mota & Teixeira, 2014). 

MOOC providers are usually organizations that accept, select, and endorse contributions for open 
courses (e.g. from Universities) and make them available for student enrolment and participation. 
Often, providers develop their own platforms, emphasizing their features as a means to attract 
authors and participants. In addition to traditional learning management systems, MOOC 
platforms provide novel features, such as karma points, badges, and communication tools. Most 
MOOCs offer additional, usually paid, online and offline services, such as tutoring, course 
materials, and formal certification (Hew & Cheung, 2014). 

MOOC courses have greatly grown in number and impact, and there is a large supply involving 
several of the most prestigious research universities in the USA. This movement generated 
European responses, either isolated, or in coordinated initiatives, but increasingly supported by 
National Governments and by the European Commission. One of the first European initiatives 
was Openup Ed (http://www.openuped.eu/ : supported by the European Commission), which 
consisted in the release of a regional MOOC portal with a high number of course providers in 
different languages. Universidade Aberta (UAb), the Portuguese Distance Learning University, 
recently contributed to this movement by developing a specific MOOC platform as well as an 
innovative pedagogical model for MOOC, the iMOOC (Teixeira & Mota, 2013). 

Increasing access to quality education, raising institutional visibility or building brand, increasing 
student recruitment, and improving or innovating pedagogy are the main goals of MOOCs for 
Higher Education institutions (Hollands & Tirthall, 2014). The desire to learn about a new topic 
or to extend current knowledge, the curiosity about MOOCs, for personal challenge, and the 
desire to collect as many completion certificates as possible, are the main reasons why students 
sign up for MOOCs (Hew & Cheung, 2014). 

Typically, MOOCs not only integrate the connectivity of social networking, the facilitation of an 
acknowledged expert in a field of study, and a collection of freely accessible online resources, but 
most importantly, they build on the active engagement of “students” who self-organize their 
participation according to learning goals, prior knowledge and skills, and common interests 
(Hmoudova, 2014). 

Nevertheless, Spector (2014) argues that MOOCs are overstating the likelihood of having a 
significant and sustained impact as an educational technology. MOOCs can be used effectively to 
support existing learning and instruction, but as stand-alone environments they fall short, unless 
they have a learning guidance, formative feedback, and overall assessment. MOOCs have sparked 
an intense new debate about the role of technology in shaping higher education and they pose 
inherent challenges in trying to automate instruction for an unlimited audience. If MOOCs are to 
achieve the revolutionary potential often claimed, the critics argue that MOOC advocates may 
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want to reflect upon the origins of the movement and focus on developing and incorporating more 
creative and empowering forms of open online learning (Toven-Lindsey, Rhoads & Lozano, 2015). 

MOOCs inspire many research questions. What are their business models? What educational and 
technological challenges are posed by this type of courses? Who bears the cost of the 
infrastructure, and the development and implementation of courses? Which platforms to use? 
What is the capacity of an installation for massive numbers of students? (Hollands & Tirthall, 
2014). Should participant accomplishment in non-formal educational provision as massive online 
education be assessed using the same indicators as in formal education? In fact, completion rates 
have been extremely low in MOOCs (Alraimi et al., 2015), averaging 7 - 10%, which points to 
limitations in the design and execution of these courses (Toven-Lindsey et al., 2015; MaHew & 
Cheung, 2014; Hmoudova, 2014). But, independently of the design shortcomings of such MOOC 
provision, is the previous statement an appropriate assumption to make? 

If decision makers are to decide rationally about engaging in MOOC production, it is critical to 
know whether MOOCs are both learning and cost effective in delivering quality education 
(Hollands & Tirthall, 2014). Thus, some of these MOOC developments seem to be operating in a 
policy vacuum regarding open learning in general. At some point, institutions will need to develop 
a clearer, more consistent strategy for open learning, in terms of how it can best be provided, how 
it calibrates with formal learning, and how open learning can be accommodated within the fiscal 
constraints of the institution, and then where MOOCs, other OER, and conventional for-credit 
online courses might fit with the strategy (Bates, 2015). For example dropout rate and behaviour 
of lurkers (silent and non-active participants) in such large groups of non-formal students 
represents one of the most puzzling issues for most educators in online courses (Rodriguez, 
2012). 

The aim of this paper is to assess the participants’ activity, benefits and impacts of the novel 
approach used on the pilot course of the iMOOC developed by UAb. This particular MOOC model 
includes an integration of platforms and an adaptation and development of pedagogical 
resources, which combined features of both c and x MOOC approaches. Within this work, the 
following research questions were addressed: i) is the chosen integrated technology together with 
the pedagogical iMOOC model efficient (in terms of connectivity and individual instructor 
design)?; ii) does the institution benefit with the iMOOC development? Lessons learned from this 
pilot are discussed to help decision-makers in higher education to rationally decide on effective 
strategies to deliver MOOCs. 


Massive Open Online Courses at UAb (The iMOOC Model) 
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In 2007, UAb became a reference European institution in the area of advanced elearning and 
online learning as a result of the implementation of its Virtual Teaching Model, unprecedented in 
Portugal and developed by this institution. 

As such, the pedagogical model for MOOCs at UAb (iMOOC) was built upon the four main pillars 
of UAb’s virtual pedagogical model: learner-centredness, flexibility, interaction, and digital 
inclusion (Pereira, Mendes, Morgado, Amante & Bidarra, 2008). The iMOOC pedagogical model 
can therefore be integrated as a variant of UAb’s institutional formal elearning pedagogical 
model, a specifically designed variant for MOOCs combining autonomous and self-directed 
learning with a strong social dimension. It articulates flexibility with the pacing necessary to help 
students “getting things done” in face of their pressing everyday commitments (Teixeira & Mota, 
2013). UAb wanted to make learning available to as many as possible, but also to bring these 
people into the digital online environment, where a crucial part of modern life happens, thus 
helping curb the digital divide (Teixeira & Mota, 2014a). Most importantly, the iMOOC 
pedagogical model was designed in such a way as to facilitate the transition of students from non- 
formal education to formal education (Teixeira & Mota, 2014b). 

One of the main characteristics of the iMOOC approach is the high level of transparency of the 
learning process. In fact, registration is only required for publishing in the institutional spaces but 
all content is open access, i.e. anyone can read it. Another innovative element is the introduction 
of a first module, usually a week long, for participants to get acquainted with the spaces, tools, 
and services, as well as with the processes of work and communication that will be used in the 
course. This “familiarization” process, already present in Universidade Aberta’s Pedagogical 
Model as a standalone module that all students have to take when they start their learning at 
UAb, has proved to be an essential component in student success and satisfaction regarding their 
online learning experience (Teixeira & Mota, 2014a). 

Learning is learner-centered and based on the realization of activities. It should be evidenced 
through the creation of online artifacts (texts, videos, presentations, slidecasts, mind maps, mash- 
ups, etc.) that demonstrate the learner’s knowledge and competencies regarding the material 
studied. The learning process combines autonomous self-study and reflection with interaction 
with other participants in an open social context. Participants are expected to take an active role 
in and be responsible for their own learning, but also to actively engage in helping build a 
supporting learning community (Teixeira & Mota, 2013). 

This same integration strategy was followed in the design of the virtual learning environment. 
Instead of building a custom platform, or adapting one of the recently created (and rather 
unstable) special-purpose platforms for MOOCs (like OpenMOOC, http://openmooc.org/. Gil & 
Martin, 2013, or EdX, http://code.edx.org) . an integrated architecture combining well- 
established, well-tested software, with large bases of users was designed. Moodle 
(https://moodle.org/) and Elgg (http://elgg.org/) were chosen as components, fulfilling all 
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pedagogic and technical requirements, and having the additional advantage of sharing the same 
underlying technologies (PHP language, databases, and operating systems). 

Formal and formatted learning aspects (content access, course organization and sequencing, 
quizzes, assessment) are typically provided by learning management systems (LMS), like Moodle. 
Most elearning and online distance learning initiatives use a LMS for the development of closed, 
instructor-controlled courses. A MOOC is, by definition, an open course, and to provide an open 
environment for students to freely interact and share their opinions, thoughts, contents, web 
bookmarks, etc., a social networking platform was added, based on Elgg, integrated in a global 
environment, as seamless as possible, with Moodle. The integration was done using the IMS 
consortium specification for LTI (learning tools interoperability), in its basic version (Severance, 
2010), providing user single sign-on and profile information sharing. 

Moodle is an open-source learning management system, with over 71 million users globally, and 
used in more than 68,000 organizations all over the world, including major distance teaching 
universities. Moodle is built with “social constructionist pedagogy” in mind although it doesn’t 
force users to use that style of learning. It is organized around courses, each one with a list of 
enrolled students that typically are allowed access to that course (Moodle, 2015). In this way, 
Moodle works as a set of separate virtual classrooms, where resources are delivered and 
teaching/learning activities occur according to a plan and within the control of the instructor. For 
organizing contents and activities within an iMOOC, Moodle was chosen, as its strength lies 
precisely in providing a learning space with standard tools for displaying resources, performing 
activities, and assessing students. There are clearly distinct teacher and student roles, which 
determine what each user can do in a virtual classroom. Thus, Moodle functions both as an 
authoring tool for creating courses, and as a back office for tutor access and their interaction with 
students. 

Elgg is an open-source, social networking platform also widely used where web-based 
communities can be created and managed. The central concepts are the user and her/his relations 
with other users. Contraiy to Moodle, Elgg focuses on open interaction and promotes resource 
sharing. As MOOCs are much more student-centered than typical online learning (Hew & 
Cheung, 2014), a platform like Elgg was a natural choice for managing informal contact among 
learners, sharing resources and learning outcomes, and also for conducting formal learning 
activities in a more open way. 

In iMOOC, Elgg functions as a front office, where students register and access their courses, with 
Moodle acting as a back office (Figure 1). The BLTI connection is configured to treat Moodle as a 
learning tool provider, and Elgg as a consumer, so that students can access the platform through 
Elgg and conduct all their activities by navigating seamlessly between both systems. 

With this platform architecture, the iMOOC pedagogical model follows a mixed approach between 
the xMOOCs and the cMOOCs, aiming to gather the advantages of both MOOC types. On one 
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hand it follows the xMOOC principles by using a designed LMS software (Moodle) that allows for 
a teaching model focused on the transmission of information, with high quality content delivery, 
computer-marked assessment (mainly for student feedback purposes), and automation of all key 
transactions between participants and the learning platform (Bates, 2014). On the other hand, by 
integrating in the Elgg social networking platform, the iMOOC environment allows for a strong 
emphasis on networking and in particular on strong content contributions from the participants 
themselves, according to the cMOOC principles. Indeed the principles of cMOOCs, like autonomy, 
diversity, interactivity, and openness, to allow the sharing of knowledge between participants 
(Bates, 2014) are put into practice in the iMOOC model. 



Tutor 


Figure 1. iMOOC platform architecture. 


The Pilot iMOOC on "Climate Change: The Context of Life 

Experiences" 

The pilot course “Climate Change: the context of life experiences” was designed according to the 
iMOOC model. This course was chosen as the first MOOC experience at UAb, due to the broad 
interest of climate change issues, and the importance of education and engagement of the general 
public as way forward to minimize and adapt to this worldwide environmental problem. 

The aim of the course was to introduce the concept of climate change in the context of sustainable 
development, relating it to each individual’s life experience, such as in adaptation and mitigation 
of climate change effects, related to floods, water shortages, increase of sea level, coastal erosion, 
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and changes of biodiversity. The course also aimed to help comprehending and critically 
analysing the information on climate change produced by a variety of organisations, under a 
diversity of perspectives, including its scientific, political, economic, and social perspective. The 
pilot course had a duration of 8 weeks, from the May 6 th until July 1 st , 2013, and was preceded by 
a five-day “boot camp” module. This component is quite a distinctive and unique feature of the 
iMOOC model as well as UAb's virtual pedagogical approach. The “bootcamp” module is meant 
for participants to get acquainted with the spaces, tools, and services, as well as with the processes 
of work and communication that will be used in the course (Teixeira & Mota, 2014b). Participants’ 
effort was 4 ECTS (European Credit Transfer System), but the course remained open after 
learning activities had been completed. 

Two main learning tools were used: (1) the "Learning Guide" containing all the information 
concerning the course, which included course objectives, syllabus and developed skills, 
pedagogical model, learning resources, timetable of learning activities; and (2) the forum "News 
and Headlines" for publication of messages relating to course which established the main weekly 
study guidelines and feedbacks. The course was structured in 5 sequential topics: 1. What science 
tell us about climate change; 2. Economy and climate changes: 3. Politics of climate changes; 4. 
The sociological perspective of climate changes and 5. Perspectives integration within sustainable 
development. To ensure instructional quality, according to Margaryan, Bianco & Littlejohn 
(2014), several strategies were applied: each topic had an introductory short video presented by 
the course teachers, a summary of suggested activities and a document with detailed guidelines 
for their development, as well as learning materials (OER, in the iMOOC model) and other 
support materials. Each topic also had a discussion forum where participants could interact with 
each other. Weekly feedback by the teachers was drawn upon the tutors’ synthesis of subjects and 
debates on the Moodle forum and on the Elgg tweets and blog messages of the participants. A 
video chat conference using Google Hangout (https://plus.google.com/hangouts) was conducted 
with an invited expert on climate change and the course teachers, moderated by the tutors, as 
another way to engage participants. 

The participants were expected to study independently, exploring resources, searching and 
sharing for other relevant learning material, conducting activities and reflecting on their learning 
experience, also contributing themselves for the production of learning materials (texts, videos, 
presentations, mind maps, etc.), i.e. demonstrating their understanding of the topics and skills by 
producing artifacts as evidences of their successful learning. Participants were also expected to 
become actively involved in the interaction with other participants and to assume an active role in 
dialogue on the issues under study and contribute significantly to the knowledge being built, 
creating in this way a dynamic and collaborative community. 

Upon completion of the course, the participants who wished to obtain a Certificate of Completion, 
had to be positively evaluated in the two course assignments, as defined in the course Learning 
Guide. Assignments were evaluated by a process of peer review (by three participant reviewers), 
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based on a criteria matrix provided by the teachers responsible for the course, and through the 
use of the Workshop tool, provided by Moodle. Participants wishing to obtain a certificate of 
completion had to submit their assignment on the platform and to participate in the peer review 
process. Participants who wished to obtain formal accreditation of the course (i.e. 4 ECTS) had to 
register and to be positively evaluated on a written face-to-face exam (50% of final classification), 
as well as to be positively evaluated on a portfolio of activities developed during the course, 
assessed by the university staff (remaining 50%). As the pilot course clearly showed, the 
certification options embedded in the courses allow participants to manage their own learning 
experience, determining the kind of final output that is best suited for their learning needs. This 
includes the possibility of their non-formal learning experience being subject to formal 
accreditation (Teixeira & Mota, 2014b). 

The team involved in the planning, developing, and teaching of the course had two experts in 
computer science, for platform development; two education experts, for developing the iMOOC 
pedagogical model; two environmental sciences experts, for scientific contents and course 
organisation; three tutors for students attendance, and four multimedia technical staff, for video 
development and production. 


Methods 

In order to assess the efficiency of the chosen integrated technology for the iMOOC, all events 
registered on both Elgg and Moodle platforms were analysed, to learn about the profile of the 
pilot iMOOC participants and to assess their active participation. Indicators for ’’pageviews”, 
’’users” and ’’posted messages” were collected in three timescales - day, month, and year, from 
January 1 st 2013 until January 1 st 2014, allowing an analysis of activity before, during, and after 
the course (since it remained open for visitors). A ’’pageview” occurs each time a user clicks on a 
link. The “users” indicator measures the number of users that access the course in a certain time 
period. The “posted messages” (both in Elgg and Moodle) count public messages, and is an 
indicator of participants’ involvement, as active users. Daily data were gathered for specific 
activities, to determine degree of participation, and message types were separated to find evolving 
patterns of behavior throughout the course. Since some users are more active than others, we also 
cross number of events and messages with the number of activity days, to distinguish sporadic 
participants from the ones that regularly follow the course (allowing us to assess the participants’ 
profile). Analysis of the number of visualisations of resources and activities throughout the course 
was also conducted and charted. 

To evaluate the benefits to the University with the development of iMOOC, as well as its 
effectiveness as an educational tool, participants success was assessed by several indicators, the 
most traditional and formal one being completion rate. This was calculated on the basis of the 
number of participants with positive evaluation in the assignments, and face-to-face exam and 
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portfolio. However, as stressed earlier, this is not a reliable indicator for non-formal modes of 
education delivery as MOOCs. The benefits for UAb were also assessed by analysing if iMOOC 
contributes to increase student recruitment, one of the aims that literature identifies as a driver 
for Higher Education Institutions involvement on MOOC provision. For that purpose, we tracked 
participants in the iMOOC that also enrolled in UAb formal courses, during the 18 months after 
completion of the iMOOC on Climate Change. Cost analyses of delivery of the pilot iMOOC were 
conducted using the ingredients method (Levin & McEwan, 2001) to estimate the costs of 
production and delivery of this pilot iMOOC. Personnel costs were assumed to represent 75% of 
total costs of the educational interventions (Levin & McEwan, 2001) and were calculated based on 
the salaries for each job category of the team personnel. Cost of re-running the iMOOC was also 
estimated. 


Results 

Analysis of the Platforms' Events and Participant Activity 

The course was advertised for two weeks, resulting in the enrolment of 1,024 participants. Further 
enrolments were blocked at this point, to ensure platform stability, and considering that this was 
a pilot experience. 

The analysis of participant-generated events during a day in both platforms shows that most 
course activities occur between 9 AM and 11 PM, i.e. during daytime (Figure 2). 



Hour of Day 


- PV Moodle 

-USR Moodle 

- MSG Moodle 

- PVEIgg 

- USR Elgg 

MSG Elgg 
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Figure 2. User events in both platforms by hour of day (PV - pageviews; USR - users; MSG - 
posted messages). 

In addition, the number of users in Moodle and Elgg were similar throughout the day, indicating 
that most users that accessed Elgg, also entered Moodle through the integrated environment (see 
Figure 2). However, differences exist in pageviews: during the high activity period (from 9 AM 
until 11 PM) there are more page views in Elgg than in Moodle, while the opposite occurred in the 
low activity period. According to the number of posted messages, posting activity is always higher 
in Elgg than in Moodle, even though the number of users logged in is similar. In fact, messages 
are more easily posted in Elgg than in Moodle, since Elgg is a well-known social/collaborative 
platform. 

Figure 3 shows the activity on the two platforms on a monthly basis. The course ran during 
months 5 and 6 (May and June), so the amount of actions before (course preparation and two 
week online registration), and after the course (visiting users) is much lower, compared to the 
period of the course proper. 



-PV Moodle 

- USR Moodle 

- MSG Moodle 

- PV Elgg 

USR Elgg 
MSG Elgg 


Figure 3. Monthly user events in Moodle and Elgg before, during (months 5 and 6) and after the 
course (PV - pageviews; USR - users; MSG - posted messages). 


During the course, the number of pageviews in Elgg was higher than in Moodle. After the course 
ended the opposite situation was observed: the number of pageviews in Moodle was consistently 
higher than in Elgg. This appears to confirm that during the course, interaction through social 
networks is key to drive the learning process and is also at the center of visitors’ attention, but 
after the course finishes, visitors focus on course materials rather than on content produced on 
the social media platform. 

This work is licensed under a Creative Commons Attribution 4.0 International License. 


162 













iMOOC on Climate Change: Evaluation of a Massive Open Online Learning Pilot Experience 
Rocio, Coelho, Caeiro, Nicolau and Teixeira 


The daily analysis of events in both platforms during the course (Figure 4) shows, as before, that 
almost all users that entered Elgg also entered Moodle, but viewed more pages in Elgg. The 
activity in both platforms increased in the first days (from May 6 th ) of and then progressively 
lowered until the end of the course (July 1 st ). Nevertheless, several peaks of activity, mainly as 
Elgg pageviews (and less as Moodle pageviews) were observed during quizzes and assignments 
weeks (quizzes: May 21 st to 27 th and June 11 th to 17 th ; assignments: June 4 th to 10 th and 18 th to 
24 th ), and the expert video conference (June 20 th ). Instructional messages of motivation 
conducted by teachers were also posted at the beginning of the week, when those peaks were 
observed. 



-PV Moodle 
-USR Moodle 
-MSG Moodle 
PV Elgg 
USR Elgg 
MSG Elgg 


Day of Month 


Figure 4. Daily user events in Moodle and Elgg during the course (PV - pageviews; USR - users; 
MSG - posted messages). 


Profiling the iMOOC participants in terms of number of pageviews and posted messages, 
accounted for both Moodle and Elgg (Table 1), we observed that all 246 users with one pageview 
didn’t post any messages, hence dropping out very soon. Similarly, most of the participants with 
2-10 pageviews didn’t post any messages. Participants with the most number of posted messages 
were also the ones with most pageviews (although there were 40 participants in the group of 101- 
1000 page views with no posts). Globally, most of the pilot iMOOC participants (867, i.e. 85%) 
didn’t post any messages, regardless of pageviews. 
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Table 1 

iMOOC participants in terms of numbers ofpageviews and posted messages (both platforms). 


Pageviews 

Posted messages 

0 1 2-10 

11-100 

Total users 

1 

246 




246 

2-10 

316 

3 



319 

11-100 

264 

57 

14 


335 

101-1000 

40 

21 

47 

9 

117 

> 1000 

1 

1 


5 

7 

Total users 

867 

82 

61 

14 

1024 


Analysis of the participant profile of the course, in terms of activity days with pageviews and 
posted messages (Table 2), shows that participants with less than two days of pageviews didn’t 
post any messages. Participants who were more active during the course fall into the groups 
shown in bold in Table 2, i.e. the groups with greater global number of users (respectively 39 and 
469). Still, a considerable number of participants (425) were active in terms of page viewing (11- 
1000) but did not post messages. 

Table 2 

iMOOC participants in terms of the number of days of pageviews and posted messages (both 
platforms combined) 



Days with posted messages 


Days with pageviews 

0 

1 

2-10 

11-100 

101-1000 

Total users 

1-2 

356 





356 

3-10 

170 

1 




171 

11-100 

414 

5 

14 

36 

469 

101-1000 

21 

1 

1 

3 

2 

28 

Total users 

961 

7 

15 

39 

2 

1024 
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Considering the activity of participants in Moodle alone, activity decreased globally from the 
beginning of the course through topic 1 and 2 (Figure 5), but stabilised thereafter until the end of 
the course, as shown by the number of visualisations of the introductory videos, assignments and 
Moodle fora. Quizzes and complementary materials were the least visualised resources, as was 
the e-textbook, that covered all topics, and whose accesses probably were in the form of 
downloads. 


15000 

12500 

10000 

7500 

5000 

2500 

0 



Course Topic 1 Topic 2 Topic 3 Topic 4 Topic 5 


■ Study guideline 

■ Forum 

■ e-Textbook 

■ Video 

■ Complementary material 

■ Quizzes 

la Assigne me nt 


Figure 5. Number of visualisations of activities/resources in Moodle, throughout the 5 topics of 
the pilot course. ‘Course’ refers to the common resources: “Learning guide”, “News and 
Headlines” forum, e-textbook, and introductory video. 


Benefits and Impact of the iMOOC Pilot 

The participants learning success on the iMOOC’s pilot on Climate Change may be assessed as the 
success rate in terms of user’s participation and of motivation to complete the training (Table 3). 
This was measured by the participants’ positive evaluation of the two assignments in the peer 
review process, as 3.1%. Even though there was a decrease in the delivery of assignments by the 
participants (and in peer review), the data appears to indicate that the four main assessment 
phases were important to retain participants throughout the course. Only 0.3% registered for the 
exam and from these, 67% were accredited for their training. 

Table 3 

iMOOC participants evaluation success 
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Main assessment 
moments 

Handled for evaluation 
(96) 

Peer review process 

Quiz 1 (auto-assessment) 

9.1 (n=93) 

- 

Assignment 1 

3.9 (n=40) 

3.6 (n=37) 

Quiz 2 (auto-assessment) 

3.2 (n=33) 

- 

Assignment 2 

3-3 (n=34) 

3.1 (n=32) 

Exam (face-to-face) 

0.3 (n=3) 

0.2 (n=2) 


Many participants (50.1%) were already UAb students, prior to their registration in the pilot 
iMOOC. However, 1.5 % of these were newly-enrolled in formal courses at UAb. Still, 48.4% were 
unidentified and may represent a potential group for future recruitment. 

If we broaden the definition of learner achievement in MOOCs to cover other objectives of 
participants who may be interested in just following the course but not participating in the 
assignments, we can calculate another success rate. In Table 2 we registered 497 
(469+28) participants with a period of activity covering the course, that represent 48.5% of the 
participants. 

Benefits to the institution may also be accounted for from the income from participants 
registering and accreditation processes (0.3% of participants), and from new enrollment in UAb 
non-formal and formal programs (1.6% of participants - 9 long life learning courses, 4 first degree 
and 3 post-graduates), which amounts to a minimum of 2,640 Euros in the pilot run, based on the 
iMOOC exam fee and UAb program’s tuition. Nevertheless, we can’t prove that these new 
enrollments were strictly and only related with iMOOC participation. 

A clearer indicator of success is the high level of satisfaction shown in the user survey. Answers to 
the questionnaire revealed that 83% of the respondents rated the course quality as excellent or 
good (Teixeira & Mota, 2014b). 


Costs of the Pilot iMOOC Production and Delivery 

Based on the analysis of costs that were conducted for the production and delivery of this pilot 
iMOOC at UAb, personnel costs were estimated as 11,124 Euros (Table 4). As assumed 
methodologically, facilities, equipment and overhead were not calculated, but should represent 
25% of the total costs (Levin & McEwan, 2001), hence the total cost of the pilot iMOOC was 
estimated as 14,830 Euros. The costs of development of the iMOOC pedagogical model and 
platform were also not included. 
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Table 4 

Personnel Costs Estimated for the Pilot iMOOC. 



Team 

members 

(n) 

Work (hours) per team 
member 

Average estimated cost 
(Euros) 

Faculty staff 




Computer science 
expert 

2 

80 

3 120 

Education science 
expert 

2 

40 

1560 

Environmental expert 

2 

76 

2 964 

Instructor/tutor 

3 

32 

1 800 

Multimedia technical 
staff 

4 

78 

1 680 

Total 



11124 


Re-runs of the course require less design and development time. The personnel cost of the course 
re-run, was estimated on the basis of the personnel costs of the three instructors (32I1 each), two 
environmental experts, and two computer science experts (respectively 20, 10 and 20 hours 
each), as 1,900 Euros. Global cost is hence estimated as 2,530 Euros. 


Discussion and Conclusions 

The predictable university’s benefits from the iMOOC production and delivery included (based on 
Hew & Cheung, 2014, Hollands & Tirthall, 2014, Hmoudova, 2014): (i) increasing participants 
knowledge and awareness on the topic of Climate Change, (ii) raising of the institutions’ profile 
and visibility, (iii) increasing student recruitment, and most importantly, (v) improving or 
innovating pedagogy. 

The success rate obtained in the pilot iMOOC developed at UAb, accounted for as number of 
participants who were issued a completion certificate, was within the usual low values reported 
for MOOCs (3.1%). Nevertheless 45.8% of the participants were active during the course, 
suggesting that MOOCs can be useful learning opportunities, though not necessarily for getting 
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the final approval and certificate. On this case study the social network platform Elgg was used, 
where a strong and active participation was observed and where the periodic (weekly) feedback 
and instructional messages by the teachers and continuous monitoring by tutors was conducted. 
Also iMOOC participants frequented a five-day “boot camp” module prior to the course for 
familiarization with the online environment and for initiating social bonding, following the best 
practices specified on Pereira et al. (2008). Furthermore, on the pilot iMOOC, the use of 
introductory videos in each topic and a web conference were also important tools as participant 
motivators. Margaryan et al. (2015) randomly selected a sample of MOOCs and concluded that 
although most MOOCs are well-packaged, their instructional design quality was low which is an 
important variable in the overall quality of a course. These authors advised for the application of 
instructional design principles (such as integration and activation) in particular, since MOOCs are 
still in the experimental phase. So, the use of instructional measures and activities could justify 
the high active participation rates found in the pilot iMOOC. Also, it is well known that in online 
distance learning, the establishment of a strong sense of community among learners encourages 
higher levels of satisfaction, persistence, engagement, and cognition (Toven-Lindsey et al., 2015, 
Garrison, 2011, Morris, Finnegan & Wu, 2005). The iMOOC appears to be able to articulate this 
with a massive and open environment, thus managing to scale up online collaboration between 
learners. In addition the common design features of cMOOC (Bates, 2014) were guaranteed in the 
iMOOC model, based on learning tools, instructions, assessment, and materials. 

The peer assessment process was used for students’ assignments evaluation, using the workshop 
tool in Moodle. Peer assessment potentially enables large classes to offer assignments that are 
impractical to grade automatically. Furthermore, human grading more easily provides context- 
appropriate responses and better handles ill-specified constraints (Kulkarni, Wei, Le, Chia, 
Papadopoulos, Cheng, Roller & Klemmer, 2013). However, peer assessment imposes a particular 
schedule on class and limits student flexibility, which could explain the low number of 
participants that deliver and peer-assessed the assignments. Nevertheless, this assessment 
process is under several research improvements as described in Kulkarni et al. (2013). 

The analysis of this pilot iMOOC experiment suggest than iMOOC courses would have higher 
completion rate if their duration was shorter. This is hinted by the decreased activity of 
participants along the course. If only one peer review activity existed, more participants might 
find the time needed to make it. In accordance the iMOOC model in its current version has 
decreased the suggested duration of the courses to six weeks. However, as pointed out, and as the 
experience of UAb and other MOOC institutional providers proves, learner satisfaction and, most 
importantly, the success of the learning experience, cannot be measured by completion rates. This 
cannot be a quality indicator for non-formal open forms of education delivery because 
participants in MOOCs are typically non-homogeneous groups of learners with quite different 
backgrounds, expectations and ultimately aiming at different learning outcomes (Hernandez et 
al., 2014). 
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About 1.6% of new students enrolled in UAb, both on formal and non-formal programs, had been 
registered participants for the first time at UAb with the pilot iMOOC. Furthermore, the pilot 
iMOOC served for raising institutional visibility. Perceived reputation and perceived openness are 
the two strongest predictors to explain MOOCs continuance intention to use (Alraimi & Ciganek, 
2014). Nevertheless, individual instructors designing of MOOCs need to make sure that the 
design is consistent in terms of educational philosophy, and be clear as to why they are opting for 
a MOOC rather than a conventional online course (Bates, 2015). 

Overall the cost of developing and delivering this pilot iMOOC at UAb vaiy between 14,830 Euros 
(for the pilot run) and 2,530 Euros (for a course re-run). This global costs can be estimated lower 
if we account for registering and accreditation processes of the participants and for student 
recruitment, which was estimated as a minimum of 2,640 Euros. Also those costs are internal, 
since no team member received any financial compensation and the course was developed in the 
framework of a research project. In fact, staff costs have been calculated as a part of salaries of the 
university staff, researchers and students work, university facilities and equipment, and no 
external funding was used for course production and delivery. Nevertheless, Hollands and Tirthali 
(2014) argue that MOOC production is often more costly than the development of regular online 
courses, the ability to scale MOOCs and the absence of associated student supports results in a 
dramatically lower cost per completer. Considering that MOOCs can help achieve other objectives 
not generally addressed by regular online courses, including branding, global reach, and large 
scale research, MOOCs would appear to be a wise use of resources, if only the costs could be 
recovered through tuition or other fees. However, as stated previously, it is arguable that course 
completion per se is not a satisfactory measure of effectiveness and that MOOCs should be judged 
on the quality and quantity of learning that takes place. 

The lack of a clearly defined business model associated with MOOCs, or at least those run for 
profit, could lead to their demise in the near future. This could be significant for universities 
which view joining the MOOC party as a way of dealing with shrinking budgets and rising costs 
(Hmoudova, 2014). Bernhard, Bittle, Van Der Vlies, Bettoni & Roth, (2013) proposes a new 
business model where the learning process of all the participants can be enriched, and the 
services provided by teachers can be funded through a specific payment system. 

Other assessments to evaluate the real effectiveness of this pilot and possible re-runs should be 
conducted. Other learning analytics can also be used to track better online learners more likely to 
drop out. Tabaa and Medouri (2013) proposes a specific Learning Analytics System for MOOCs 
that analyses, with low latency, students’ behaviour with the intent of increasing the impact of 
analytics on teaching and learning in such environments. In addition, longitudinal studies 
tracking post-MOOC outcomes such as sequences of courses taken, professional certifications 
obtained, or job opportunities received would help assess the longer term economic value of 
participating in these courses and allow for cost-benefit analyses to estimate the overall returns to 
society of investing in MOOC creation (Hollands & Tirthali, 2014). 

This work is licensed under a Creative Commons Attribution 4.0 International License. 


169 



iMOOC on Climate Change: Evaluation of a Massive Open Online Learning Pilot Experience 
Rocio, Coelho, Caeiro, Nicolau and Teixeira 


Nevertheless, several key challenges of teaching MOOCs are still on the table, namely: difficulty in 
evaluating students’ work; having a sense of speaking into a vacuum due to the absence of student 
immediate feedback; being burdened by the heavy demands of time and money; and 
encountering a lack of student participation in online forums. Issues that have yet to be fully 
resolved are the quality of MOOC education, and the assessment of student work (Hew & Cheung, 
2014). Research needs to come up with theoretical underpinnings that will explain factors related 
to social aspects in MOOCs that have a completely new context and offer practical guidance of 
course design and instruction (Gasevic, Kovanovic, Joksimovicm & Siemens, 2014). The design of 
MOOCs continues to evolve, with all kinds of variations and challenges to address, e.g. Yousef, 
Chatti, Schroeder, Wosnitza & Jakobs (2014) discuss the use of other MOOCs, such as sMOOCs 
that represent small open online courses and bMOOCs that represent MOOCs that are blended 
with on-campus teaching. 

In this paper we presented the largest Portuguese MOOC to date, on the topic of climate change, 
with more than one thousand enrolments. This course piloted what appears to be a successful 
innovative pedagogical model - the iMOOC, where the advantages of c and x MOOC were 
successfully joined, such as collaborative group work, paced instruction, and a creditable online 
course. It was clear from the technological point of view, the advantage of using open source 
platforms, reducing the cost of development, and having at the same time, a large set of stable and 
well tested features. The technology aspects were never a constraint factor. 

The further development of the iMOOC experience has been assured by its influence in other new 
hybrid approaches emerging in Europe, as the sMOOC model developed in the framework of the 
ECO project (Brouns et al., 2014). Inspired by the iMOOC, the pedagogical approach used in the 
sMOOC not only is learner-centred and promotes independent learning, as also supports adaptive 
learning strategies and ubiquitous, pervasive, and contextualized learning. As a result of this, the 
courses have the potential to adapt to the changing intentions of participants during its duration. 

Moreover, UAb's participation in a number of leading EU-funded projects, namely the above 
mentioned ECO (http://ecolearning.eu) and EMMA (http://www.europeanmoocs.eu), has 
secured resources and funding for medium-term improvement of the iMOOC experience in terms 
of technology, pedagogies, and content development, and also in the definition of appropriate 
business models. Future long term research is indeed needed to answer the various questions 
highlighted in this work. 
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